Bevilacqua 

Research 

Corporation 


University  of  Cognitive  and 

Alabama  -  Behavioral  Systems 

Huntsville 


Adaptive  Instructional  Systems  (AIS) 


Final  Report 

Contract  Number  DASW01-0OM4086 
July 23, 2000 

BRC  -TR  - 0040 - 00/001 


Prepared  By: 

David  J.  Skipper,  Ph.D. 

Harry  Delugach,  Ph.D. 

Debra  Evans,  Ph.D. 

Bevilacqua  Research  Corporation 
P.O.  Box  14207 
Huntsville,  AL  35815 
Voice:  (256)  882-6229  ext  105 
Fax:  (256)  882-6239 

Prepared  For: 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Rotary  Wing  Aviation  Research  Unit 
Building  5100 

Fort  Rucker,  AL  36362-5354 

Approved  for  public  release;  distribution  unlimited. 


This  report  is  published  to  meet  legal  and  contractual  requirements  and  may 
not  meet  ARI’s  scientific  or  professional  standards  for  publication 


20000809  085 


DnCQUAUTiriH£PSciED4 


REPORT  DOCUMENTATION  PAGE 

ftum  Approved 

DMB  Ato.  0704-0738 

{*udtc  r«tinitr<(  feuieatri  ter  llti*  oni  actum  <uf  U.  ws£*mil*ii  l.o  *v***jii  t  hour  v»t  kv^kcinq  ll«*  Unw  tot  nudmultom,  s*«u-*h*Hj  Mitten  iftilii  kcuicw*. 

{pirwtmn  ard  m»***Jrm*t  thfr  Until  ihWhImL  «M»  eutntjttrtifK*  mtui  t*okv*#*t  M*»  culweSam  at  infcrmtfian.  3#r*l  ctj*w»it»fc»  Ifcgttdifttl  **»  tucMfi  niton  *t*  a*-  wrtv  «<#«ar  Mmwul  to*  IN* 

«*j|«*dfc4N «*  .ttcluiitev* lur  ftidUiitttj  Hi*  iwttoK,  io  Wneifinqlnn  HWuifuaitwik  S*«V5fe«.#,  Dii*«Sw.*U*  for  &&mnrtie«t  <.^»*talnmi  uttd  ifwfKwti,  5=21  S 

0»vu  Hk*w*r,  auto  1204,  VA  22202-4302,  Mdiaita  Qtik*  «f  iSft*UKwm*M  and Stota*.  Pw*KY0<lt  fttiiucttat  (07C4-01  fitt.  Whtfhfltott,  QC  20503. 

1.  AGENCY  use  ONLY  /Lesi/e  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND 

23  Julv  2000  24  January  2 

DATES  COVERED  Hnal, 

000 -23  Julv  2000 

A.  TITLE  AND  SUBTITLE 

Adaptive  Instructional  Systems 

Final  Report 

S.  FUNDING  NUMBERS 

s.  AUTMoms)  David  J.  Skipper,  Ph.D. 

Harry  Delugach,  Ph.D. 

Debra  Evans,  Ph.D. 

7.  PERFORMING  ORGANIZATION  MAME'SI  AND  ACDRESSIESI 

Bevilacqua  Research  Corporation 

P.O.  Box  14207 

Huntsville,  AL  3581 5 

3.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

BRC-TR-0040-00/001 

9.  SPONSORING  f  MONITORING  AGENCY  NAME  (SI  AND  ADDRESSlES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Rotary  Wing  Aviation  Research  Unit 

Bldg  5100 

Fort  Rucker,  AL  36362-5354 

10.  SPONSORING  f  MONITORING 

AGENCY  REPORT  NUMBER 

1 1 .  SUPPLEMENTARY  notes 

12a.  DISTRIBUTION  ( AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited 

12b.  DISTRIBUTION  CODE 

13,  ABSTRACT  tMaxignum  200  wwdzl 

Adaptive  Instructional  Systems  are  those  systems  which  guide  the  students  instruction  based 
on  the  student’s  performance  and  current  abilities.  This  report  describes  an  Adaptive 

Instructional  System  designed  on  using  Conceptual  Graphs  to  maintain  an  internal  concept  of 
both  the  student  and  the  instructor.  The  graphs  are  shown  as  being  capable  of  monitoring  and 
guiding  the  instruction  adaptively. 

14  SUBJECT  TERMS 

Adaptive,  Trainer,  Conceptual  Graphs,  Learning, 

1 5.  NUMBER  OF  PACES 

1 6.  PRICE  CODE 

i  ?.  security  classification  ts.  security  classification  m  sseuitiTY  classification 

O pmpcm  OF  THIS  PAGE  OP  ABSTRACT 

Unclassified  Unclassified  Unclassified 

20.  LIMITATION  DP  ABSTRACT 

Unlimited 

NSM  ?S*<KH-2aQ-$$G0 


Stnrseiard  Form  293  Iftav,  2>&ti 
described  by  ANSI  Std.  Z3SM3  29&102 


UQAFPCVVOO 


I 


Table  of  Contents 

Contents 

Introduction .  1 

Research  Perspective .  1 

Research  Goals .  2 

Research  Issues .  2 

Research  Team .  2 

Technical  Approach .  3 

Significant  Results .  4 

Document  Structure .  4 

Section  Descriptions .  4 

Appendices . .  4 

Study  Overview .  5 

Adaptation  Description .  5 

Previous  Adaptive  T rainer  Work .  6 

Trainer  Design  Approach .  7 

Student  and  I  nstructor  Conceptual  Models .  7 

Evaluating  Performance  with  Conceptual  Models .  7 

Adapting  Conceptual  Models .  8 

Learning .  8 

Implementing  Conceptual  Models .  8 

Findings .  9 

Design  Overview .  10 

Instructor  Model .  10 

Student  Model .  14 


Training  Analyst 


16 


Training  Manager .  17 

Presentation  Manager .  17 

Summary .  18 

References .  19 

Conceptual  Modeling .  A-1 

Potential  Training  Task  for  Student .  A-1 

Current  Situation  at  Start  of  Task .  A-1 

Task . .  A-1 

Student  Profile .  A-2 

Other  Impacts  to  the  Model  Contents .  A-3 

Impacting  Simulation  Factors .  A-4 

Impacting  Pedagogic  Factors .  A-4 

Example  Operation .  A-5 

Student  Mental  Model  During  Step  1  of  Task .  A-5 

Example  Adaptive  Instruction .  A-8 

Example  Task .  A-9 

Waypoint  A .  A-9 

Waypoint  B .  A-14 

Current  situation  -  Waypoint  B .  A-14 

Adaptation .  A-1 7 

Index .  1-1 

Figures 

Figure  1  Adaptation .  6 

Figure  2  System  design .  1 1 

Figure  A-1  Waypoint  sequences .  A-9 

Figure  A-2  Assumed  mental  model  -  waypoint  A .  A-10 

Figure  A-3  Planned  instructor's  direction  to  student  -  waypoint  A  A-10 

iv 


Figure  A-4  Expected  student  behavior  -  waypoint  A .  A-1 1 

Figure  A-5  Expected  resulting  situation  -  waypoint  A .  A-1 1 

Figure  A-6  Observed  student  behavior  -  waypoint  A .  A-1 1 

Figure  A-7  Expected  situation  -  waypoint  A .  A-1 2 

Figure  A-8  Inferred  mental  model  -  waypoint  A .  A-1 2 

Figure  A-9  Processing .  A-1 3 

Figure  A-1 0  Assumed  mental  model  -  waypoint  B .  A-1 4 

Figure  A-1 1  Planned  instructor's  direction  to  student  waypoint  B .  A-1 4 

Figure  A-1 2  Expected  student  behavior  -  waypoint  B .  A-1 5 

Figure  A-1 3  Expected  resulting  situation  -  waypoint  B .  A-1 5 

Figure  A-1 4  Observed  student  behavior  -  waypoint  B .  A-1 5 

Figure  A-1 5  Inferred  mental  model  -  waypoint  B .  A-1 6 

Figure  A-1 6  Processing .  A-1 6 

Figure  A-1 7  Adaptation . .  A-1 7 


Section 

1 


Adaptive  Instructional  System  (AIS) 

Overview 


Introduction 


Research  Perspective 

Warfare  in  the  middle  of  this  century  was  based  on  simpler  technology.  In  past  conflicts, 
the  war  fighters  used  limited  local  data.  The  combat  was  direct  and  they  learned  to  use  their 
combat  systems  by  rote  so  that  they  could  function  effectively  even  when  highly  stressed  in  a 
combat  environment.  These  combatants  also  needed  highly  developed  individual  perceptual 
motor  skills  for  controlling  and  aiming  the  weapon  systems  in  use  at  mid-century.  That  weapon 
technology  has  been  extended  by  information  technology,  which  brings  a  broader  scope  of  the 
battle  to  the  war  fighter  earlier  in  the  conflict.  This  permits,  and  requires,  the  war  fighter  to 
manage  the  presented  information  and  to  think  about  the  situation  and  courses  of  action  required 
by  a  detailed  opponent  picture.  The  breadth  and  depth  of  information  requires  not  only  higher 
level  reasoning  in  the  individuals,  but  also  effective  coordination  of  war  fighters  on  a  larger  scale 
than  with  traditional  crew  served  weapon  systems. 

While  technology  was  revolutionizing  information  usage  in  war,  the  high  cost  of 
personnel,  and  the  limited  budgets  in  the  latter  part  of  the  century,  decreased  the  resources 
available  to  train  the  war  fighters.  These  reductions  in  resources  occurred  at  a  time  when  large 
numbers  of  limited  intervention  missions  were  assigned  to  the  services.  This  further  stressed  the 
remaining  resources,  so  that  training  the  advanced  war  fighter  suffered. 

Even  when  training  resources  can  be  found,  traditional  training  methods  are  deficient  for 
the  advanced  war  fighter’s  training  needs.  A  major  shortcoming  is  the  “mass  production” 
concept  behind  military  training.  With  everyone  in  lock  step,  the  slower  students  set  the  pace  for 
everyone.  Tailored  instruction,  which  pushes  faster  students  through  the  training  courses,  could 
speed  up  training,  but  is  very  complex  for  a  training  staff  to  administer.  Therefore,  only  small 
groups  can  capitalize  on  this  approach.  Small  groups  require  more  instructors,  which  is  labor 
intensive.  Tailored  instruction  appears  to  become  infeasible  for  traditional  training  commands. 

The  solution  is  to  look  at  how  training  is  done,  then  invest  in  cost  effective  improvements 
to  the  training  process.  Intelligent  computerized  training  systems  offer  an  effective  automated 
approach  to  training.  The  intelligent  system  approach  must  provide  both  mechanical  skill 
training  and  knowledgeable  tutoring  in  modem  warfare  for  both  individuals  and  groups.  Further, 
automation  must  be  adaptive,  in  order  to  tailor  the  training  to  individual  students.  While  all 
levels  of  instruction  should  be  addressed,  the  highest  payoff  may  occur  in  basic  instruction 
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classes.  Here  large  numbers  of  students  and  instructors  mean  the  commitment  of  significant 
resources.  Sufficient  intelligence  must  be  embedded  in  an  intelligent  trainer  to  relieve  the 
instructors  from  basic  instruction  so  that  they  can  focus  on  the  most  demanding  forms  of 
instruction.  By  utilizing  fewer  instructors  in  these  basic  courses,  the  total  administrative  and 
planning  costs  for  the  instruction  is  lower  and  the  initial  investment  cost  can  be  repaid. 

Developing  a  competent  level  of  intelligence  in  a  trainer  requires  the  combining  of  a  deep 
understanding  of  the  instructional  area,  of  learning  psychology,  and  of  student  cognitive 
processes.  Clearly,  the  development  of  conceptual  models  of  the  students,  of  instructional 
methods  and  of  instructors  is  the  key  to  this  entire  process.  Therefore,  the  development  of 
conceptual  models  is  the  central  theme  of  this  report. 

Research  Goals 

The  goals  of  this  research  are  to  develop  an  intelligent  flight  trainer  design  that  can 
improve  the  quality  and  speed  of  training  with  reduced  human  intervention  and  supervision. 
Specifically  the  trainer  must: 

1)  Adapt  the  instruction  to  fit  the  student’s  performance  and  the  student’s  style  of  learning, 

2)  Within  limits,  identify  and  plan  the  student’s  instruction, 

3)  Provide  pre-flight  briefings,  in-flight  monitoring,  and  post-flight  evaluation, 

4)  Utilize  existing  simulator  hardware  and  software, 

5)  Provide  the  data  extraction  and  storage  to  support  experimentation  on  teaching  and  learning 
techniques, 

6)  Provide  a  path  to  learning  systems  that  automatically  learn  to  teach. 

Research  Issues 

Although  the  overall  goal  of  this  effort  is  to  develop  an  approach  and  a  design  for 
dynamically  adaptive  instructional  system,  which  is  sensitive  to  a  student’s  learning  style, 
learning  rate,  and  overall  strengths,  the  research  is  focused  on  specific  issues.  Three  issues  are 
identified  and  addressed  as  the  first  step  to  achieving  this  goal: 

1)  Can  a  conceptual  model  of  a  student’s  knowledge  be  defined  and  captured  dynamically? 
Similarly,  can  the  Instructor’s  knowledge  and  the  pedagogical  knowledge  be  modeled? 

2)  Can  a  computer  model  be  developed  which  efficiently  implements  a  student’s,  an  Instructor’s 
and  pedagogical  conceptual  models? 

3)  Can  a  student’s  conceptual  model  be  made  to  adapt  as  the  student  learns  or  changes 
behavior  based  on  a  knowledge  of  instructional  methods  and  Instructors? 

Research  Team 

Dr.  David  J.  Skipper,  Program  Principal  Investigator,  Bevilacqua  Research  Corporation 
(BRC) 
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Dr.  Skipper  has  more  than  23  years  experience  in  management  and  all  aspects  of  the  full  life  cycle 
for  software  engineering  design  and  development.  Dr.  Skipper  received  his  BS  in  Physics  with 
Honors  from  the  University  of  Texas  at  Arlington,  in  Arlington  Texas  and  his  Ph.D.  in  Physics  from 
the  University  of  Missouri  in  Rolla,  Missouri.  His  experience  includes  embedded  real  time  systems, 
telemetry  systems  and  data  reduction,  expert  systems,  computer  and  network  modeling,  client 
server  database  design,  computer  user  interfaces,  and  weapon  system  modeling.  At  BRC,  Dr. 
Skipper  developed  intelligent  software  modules  for  models  such  as  ITEMS  and  ModSAF  using 
advanced  techniques  in  artificial  intelligence. 

Dr.  Skipper  is  also  a  Part  Time  Lecturer  at  the  University  of  Alabama  in  Huntsville  in  the  Computer 
Engineering  Department;  His  teaching  areas  are  data  structures,  and  C/C++  programming. 

Dr.  Harry  S.  Delugach,  The  University  of  Alabama  in  Huntsville,  University  of  Alabama  - 
Huntsville  (UAH) 

Dr.  Delugach  is  a  tenured  Associate  Professor  of  Computer  Science  at  the  University  of  Alabama 
in  Huntsville  and  an  internationally  renowned  expert  in  conceptual  modeling.  His  research  interests 
are  in  the  areas  of  software  engineering,  software  requirements,  object-oriented  modeling, 
knowledge-based  systems,  conceptual  graphs,  computer  science  education.  His  formal  education 
includes  a  Ph.D.  in  Computer  Science,  University  of  Virginia,  Charlottesville.  Dissertation  Title:  “A 
Multiple  Viewed  Approach  to  Software  Requirements,”  an  MS  in  Computer  Science,  University  of 
Tennessee,  Knoxville  and  a  BA  in  Chemistry,  Carleton  College,  Northfield,  Minnesota.  He  has 
extensive  professional  experience  ranging  from  teaching  to  computer  operations  and  is  widely 
published  in  the  field  of  conceptual  modeling. 

Dr.  Debra  Evans,  Cognitive  and  Behavioral  Systems  (CABS) 

Dr.  Debra  C.  Evans  is  the  owner  and  founder  of  Cognitive  and  Behavioral  Systems,  which  is  a 
woman  owned  consulting  firm  dedicated  to  identifying  and  solving  human  performance  problems 
for  government  and  industry  customers.  Dr.  Evans  has  over  fifteen  years  of  experience  in  applied 
human  factors  and  ergonomics,  training  research,  analysis,  design,  and  production.  Her  research 
interest  and  training  are  in  learning  and  cognitive  processes.  She  has  applied  her  knowledge  and 
skills  in  these  areas,  as  well  as  her  extensive  experience  in  experimental  design  and  analysis,  to 
solve  human  factors  and  training  research  application  problems.  Her  formal  education  includes  a 
BA  from  Carleton  College,  1976,  with  an  MA  and  Ph.D.  in  Experimental  Psychology  from 
University  of  California,  Davis,  1978  and  1983  respectively.  She  has  worked  extensively  with  both 
commercial  and  government  clients,  in  many  different  research  areas  relating  to  human 
performance  issues,  whether  software,  training,  or  ergonomic  problems. 

Additionally,  Dr.  Evans  is  a  visiting  lecturer  in  Human  Factors  and  Human  Information  Processing 
at  the  University  of  Pittsburgh’s  School  of  Information  Science. 

Technical  Approach 

In  past  work,  the  BRC  Team  observed  that  intelligent  training  systems  depended  heavily 
on  the  development  of  a  behavior  model  external  to  the  trainer  itself.  The  trainer  maintained 
only  a  minimal  semblance  of  an  internal  conceptual  model.  This  profoundly  limits  the 
capabilities  of  adaptation  schemes  as  well  as  limiting  learning  opportunities.  It  was  hypothesized 
by  the  BRC  Team  that  a  detailed  internal  conceptual  model  set  would  lead  to  better  adaptation  as 
well  as  learning  systems,  the  BRC  Team  has  spent  considerable  effort  examining  conceptual 
models  of  intelligence  and  has  developed  tools,  systems,  and  software  based  on  complex 
conceptual  models  using  novel  technical  advances  based  on  Sowa’s  [1984]  conceptual  graphs. 
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Use  of  these  insights  and  tools  would  appear  to  lead  to  significant  advances  in  intelligent 
trainers.  This  overall  approach,  which  is  discussed  in  Section  2,  is  based  on  these  key  technical 
tasks: 

1)  Generating  conceptual  models  of  student,  instructor  and  pedagogical  knowledge, 

2)  Evaluation  of  student  performance  using  conceptual  models, 

3)  Adaptation  of  instructional  conceptual  models  to  induce  changes  in  the  student’s  conceptual 
model, 

4)  Implementation  and  utilization  of  the  conceptual  models  in  a  computerized  trainer  for  adapting 
subsequent  instruction. 

Significant  Results 

During  the  course  of  this  study,  the  following  key  results  were  obtained: 

1)  Initial  student  conceptual  models  were  created, 

2)  Initial  instructor  conceptual  models  were  created, 

3)  Methods  were  identified  to  dynamically  update  a  student’s  conceptual  model, 

4)  Methods  were  identified  to  adapt  the  instructional  conceptual  models  in  response  to  the 
Student’s  behavior. 


Document  Structure 


Section  Descriptions 

Section  2  covers  the  technical  overview  and  trainer  design  concepts  that  resulted  from  this  study. 
Section  3  covers  the  summary  and  the  recommendations  for  future  investigations. 

Appendices 

This  report  contains  the  following  Appendix: 

Appendix  A  Conceptual  Model  Demonstration. 
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Section 


Technical  Results 

System  Concept 


Study  Overview 


Adaptation  Description 

Adaptation  is  a  term  that  is  sometimes  ambiguous  in  various  studies.  In  order  to  forestall 
that  problem,  this  study  utilized  the  view  of  adaptation  described  here.  Figure  1  depicts  the 
overall  concept  of  adaptation  used  in  this  study.  The  concept  requires  some  baseline  of  the 
desired  “performance”  from  the  student  in  the  trainer.  The  student  then  interacts  with  the  trainer. 
By  some  set  of  observations  the  student’s  “performance”  is  measured  and  compared  to  the 


Figure  1  Adaptation 

baseline  “performance”.  The  difference  between  these  two  measurements  provides  a  basis  for 
selecting  the  next  interaction  with  the  student.  Clearly,  this  is  a  dynamic  ongoing  process  with 
four  key  elements.  The  first  key  element  is  the  definition  of  a  baseline  “performance”.  The 
second  key  is  the  observation  of  the  student  to  determine  the  student’s  “performance”.  The  third 
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is  the  determination  of  the  difference  or  errors  in  the  two.  The  final  key  element  is  the  decision 
process  to  select  the  next  interaction. 


The  word  “performance”  is  used  here  in  a  general  sense.  Not  only  is  the  student’s 
immediate  action  set  a  measure  of  performance,  but  the  student’s  concept  set  is  a  measure  of 
what  has  been  learned.  Therefore  when  performance  is  referenced,  both  of  these  enter  into  the 
performance  determination.  Further,  the  student  concept  set  can  be  viewed  as  including  both 
working  and  long  term  memory.  Consequently,  the  concept  set  consists  of  both  working  and 
long  term  concept  sets,  with,  (potentially,)  different  performance  evaluation  approaches  to  each. 

Previous  Adaptive  Trainer  Work 

The  field  of  intelligent  trainers  is  a  relatively  large  one,  with  initial  conceptual 
development  beginning  in  the  early  1980’s.  Consequently,  rather  than  survey  the  field, 
representative  examples  are  examined. 

Over  the  last  two  decades,  intelligent  tutors  have  been  developed  to  train  personnel  on 
may  different  types  of  tasks,  from  the  operations  of  shipboard  propulsion  plants  via  graphical 
representions  (Stevens,  et.  al,  1981)  to  medical  diagnoses  or  LISP  via  rules  (Clancey,  1983; 
Anderson  &  Reiser,  1985).  Most  of  these  tutors  have  used  similar  approaches  to  representing  the 
student  mental  model — as  a  rule  set,  although  they  have  frequently  differed  in  pedagogical 
methods. 

For  example,  STEAMER  (Stevens,  et.  al,  1981)  used  a  low  fidelity  simulation  of  a 
shipboard  propulsion  powerplant  to  present  functional  and  operational  concepts  to  naval 
personnel.  Explanations  were  supplied  to  trainees  based  on  the  types  of  errors  they  made.  Thus 
the  student  model  consisted  of  a  combination  of  an  error  or  “bug”  library  and  a  set  of  rules  for 
problem  selection. 

More  recent  tutors  such  as  the  LISP  tutor  developed  by  Anderson  and  his  associates 
(1989,  1990)  have  used  sets  of  production  rules  both  to  reflect  the  student  model  and  to  be  the 
knowledge  base  to  train.  GUIDON,  developed  by  Clancey  (1983)  used  a  similar  approach  for 
the  training  of  medical  diagnostic  procedures. 

Another  approach  to  student  modeling  as  seen  in  adaptive  trainers  is  the  comparison  of 
the  student’s  assumed  conceptual  model  with  a  model  of  expert  performance  on  the  same  task 
(see  for  example,  Fath,  1987).  However,  in  these  trainers,  usually  a  rule-based  approach  is  used 
as  well.  The  rule-based  approach  has  a  tendency  to  be  most  applicable  to  training  situations  that 
are  deterministic  and  with  well-constrained  dynamics.  There  are  no  gradations  of  “rightness”or 
possibilities  for  multiple  correct  or  incorrect  performance  paths  for  either  the  trainee  or  the 
expert. 
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Trainer  Design  Approach 

In  past  work,  the  BRC  Team  observed  that  intelligent  training  systems  depended  on 
development  of  a  behavior  model  external  to  the  trainer  itself.  The  trainer  maintained  only  a 
semblance  of  an  internal  conceptual  model.  This  profoundly  limits  the  complexity  of  adaptation 
schemes  as  well  as  limiting  learning  opportunities.  It  was  hypothesized  by  the  BRC  Team  that  a 
detailed  internal  conceptual  model  set  would  lead  to  better  adaptation  as  well  as  learning 
systems.  The  BRC  Team  has  spent  considerable  effort  examining  conceptual  models  of 
intelligence  and  has  developed  tools,  systems,  and  software  based  on  complex  conceptual  models 
using  novel  technical  advances  based  on  Sowa’s  [1984]  conceptual  graphs.  Use  of  these  insights 
and  tools  would  appear  to  lead  to  significant  advances  in  intelligent  trainers.  This  overall 
approach  is  based  on  these  key  technical  tasks: 

1)  Generating  conceptual  models  of  student,  instructor  and  pedagogical  knowledge, 

2)  Evaluation  of  student  performance  using  conceptual  models, 

3)  Adaptation  of  instructional  conceptual  models  to  induce  changes  in  the  Student’s  conceptual 
model, 

4)  Implementation  and  utilization  of  the  conceptual  models  in  a  computerized  trainer  for  adapting 
subsequent  instruction. 

Student  and  Instructor  Conceptual  Models 

In  order  to  demonstrate  the  capability  to  develop  conceptual  models  for  students,  the 
BRC  Team  participated  in  knowledge  acquisition  sessions  at  Ft  Rucker  on  14  April  2000,  with  a 
follow  up  session  on  17  May  2000.  Knowledgeable  Instructors  pilots  were  interviewed  on 
student  behavior  and  training  students.  These  notes  were  translated  into  typical  training 
scenarios.  From  these  scenarios,  basic  student  and  instructor  conceptual  models  were  developed. 
The  models  were  then  manipulated  as  if  in  an  actual  trainer  to  demonstrate  the  most  fundamental 
operation  of  the  conceptual  model  based  approach  to  trainer  development.  The  entire  process 
can  be  examined  in  more  detail  in  Appendix  A.  This  demonstrated  the  ability  to  capture  and 
utilize  initial  conceptual  models  of  students.  There  are  additional  details  required  for 
implementing  a  usable  adaptive  intelligent  trainer.  These  are  discussed  later  in  this  section. 

Evaluating  Performance  with  Conceptual  Models 

Three  performance  evaluation  methods  were  found  suitable  for  this  trainer.  The  first  is  to 
monitor  attribute  values  of  key  concepts  within  the  conceptual  model.  This  closely  mimics  the 
behavior  of  traditional  trainers,  making  it  possible  to  incorporate  performance  techniques  from 
existing  trainers.  The  second  technique  is  to  evaluate  the  graph  structure  itself.  The  topology 
and  the  existence  of  nodes  and  relations  represent  the  conceptual  model.  Graph  similarity 
measures  of  various  types  are  available  to  measure  the  nearness  of  one  conceptual  graph  to 
another.  Future  work  will  investigate  the  significance  of  using  a  given  measure  in  a  given 
situation.  The  third  technique  is  the  total  uncertainty  contained  within  a  student’s  model.  By 
observing  the  student  and  probing  the  student  with  key  questions,  an  estimate  of  the  regions  in  a 
model  where  the  student  has  uncertainty  in  concepts  or  behaviors  is  available.  By  maintaining 
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the  student  model  and  standard  models,  all  appropriately  normalized  within  a  given  belief 
system,  the  system  can  estimate  performance  based  on  residual  uncertainty  in  the  student’s 
models. 

Adapting  Conceptual  Models 

As  noted  earlier,  adaptation  in  the  trainer  means  the  selection  of  course  material  and 
interactions  in  response  to  the  student’s  demonstrated  abilities.  Based  on  the  differences 
determined  in  the  previous  section,  a  decision  is  made  about  the  instruction  to  be  presented.  This 
is  the  first  stage  of  adaptation.  A  conceptual  graph  decision  component  will  be  developed  to 
handle  this  task  based  on  standard  knowledge  engineering  techniques.  In  a  second  level 
adaptation,  this  conceptual  graph  will  adapt  itself  to  select  the  training  material  based  on 
perceived  factors  other  than  actual  direct  performance.  Factors  include  error  rates  and  fatigue 
factors,  which  may  alter  the  relational  basis  and  decision  points  within  the  graph. 

Learning 

Learning  by  the  trainer  is  not  a  requirement  of  this  study.  However,  by  representing  the 
student’s  conceptual  models,  both  desired  and  actual,  as  conceptual  graph  structures,  the 
progress  of  learning  in  a  student  can  be  established  by  the  graph  structures  sequenced  in  time. 
By  studying  the  evolution  of  the  conceptual  models  of  various  students,  as  directed  by  the 
instructor  model’s  graph  based  decisions,  it  is  expected  that  experimenters  may  be  able  to 
identify  key  points  in  the  student’s  graph  formulation.  These  points  drive  changes  in  the 
instructor  model’s  graph  structure.  This  set  of  conceptual  graph  sequences  in  the  instructor 
represents  the  sequence  of  learning  in  the  instructor  model.  Pursuing  the  conceptual  graph 
approach  will  permit  identification  of  a  set  of  instructional  approaches  that  a  trainer  must 
develop.  While  this  is  not  yet  self  learning  it  represents  a  means  to  prepare  an  instructor  to  learn 
new  methods  on  its  own.  In  the  longer  term,  an  encyclopedia  of  instructional  techniques  could 
be  developed  to  permit  the  instructor  model  to  experiment  with  best  approaches  for  students. 

Our  example  models  and  concepts  were  manually  elicited  through  conceptual  analysis  of 
interviews  involving  human  pilot  instructors.  A  key  feature  of  self-learning  is  the  capability  to 
develop  new  concepts  without  human  intervention.  One  technique  for  establishing  new  concepts 
is  formal  concept  analysis  (FCA)  as  described  by  Ganter  and  Wille  [1999]  and  others.  Formal 
concept  analysis  permits  collections  of  attribute  values  to  organize  themselves  into  clusters, 
establishing  related  sets  as  new  concepts.  We  will  be  exploring  the  use  of  FCA  in  expanding  the 
base  set  of  manually  acquired  mental  models. 

Implementing  Conceptual  Models 

The  implementation  process  contains  several  challenges.  First,  efficient  graph  algorithms 
must  be  implemented  to  create  the  student’s  conceptual.  Skipper  [1998]  has  accomplished  this 
for  basic  graphs  in  a  previous  trainer  test.  The  next  problem  will  be  to  implement  interfaces  to 
some  existing  trainer  equipment  to  sense  the  trainer  and  the  student.  The  design  summarized  in 
the  Design  Overview  Section  contains  a  presentation  module.  This  module  will  be  the  interface 
module  to  a  selected  simulation  and  it  will  have  to  be  customized  for  each  simulator  in  which 
this  is  to  be  embedded..  The  last  potential  problem  is  to  exchange  student  conceptual  models 
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with  other  researchers  for  study  or  evaluation  and  with  other  trainers  for  collaborative  training 
development.  Fortunately,  conceptual  graphs  have  an  ANSI  standard  format,  Conceptual  Graph 
Interchange  Format  (CGIF),  for  just  that  purpose. 


Findings 


This  study  began  by  identifying  the  issues  that  were  critical  to  an  advanced  adaptive 
trainer  development.  In  summary,  these  were  the  development  and  maintenance  of  key 
conceptual  models,  evaluation  of  the  student’s  “performance,”  adapting  the  course  of  instruction 
to  maximize  the  student’s  performance,  and  efficient  implementation  of  these  models.  During 
the  course  of  this  study,  the  following  key  results  were  obtained: 

1)  Initial  Student  conceptual  models  were  created  to  demonstrate  model  creation  (Appendix  A), 

2)  Initial  Instructor  sequence  conceptual  models  were  created  (Appendix  A), 

3)  Methods  were  identified  to  dynamically  update  and  evaluate  a  student’s  conceptual  model, 

specifically 

■  Performance  evaluated  by  values, 

■  Graph  similarities, 

■  Certainty  levels. 

4)  Methods  were  identified  to  adapt  the  instructional  conceptual  models  in  response  to  the 

Student’s  behavior,  such  as 

■  Fixed  selection  of  instructional  graph  templates 

■  Adaptive  selection  of  instructional  graph  templates 

These  were  used  to  develop  the  top-level  adaptive  trainer  design  presented  in  the 
following  section. 
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Design  Overview 


Figure  2  shows  a  block  diagram  of  an  adaptive  intelligent  flight  trainer.  There  are  six 
major  components  in  the  system.  The  first  is  the  presentation  manager.  Because  this  system  is  a 
software  system  designed  for  insertion  into  an  existing  trainer,  the  Presentation  Manager  serves 
as  the  interface  module  to  whatever  trainer  system  is  selected  as  the  host  system.  It  translates  the 
sequences  of  instructional  concepts  requested  by  the  instructor  into  actual  simulated  sequences 
by  invoking  the  host  simulator  system.  It  also  maintains  a  Student  Observer  module  that  utilizes 
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The  Instructor  Model  will  supply  training  to  a  student  using  the  pedagogic  methods  that 
are  the  most  effective  for  that  individual.  These  sequences  and  methods  will  be  modeled  and 
retained  as  conceptual  graphs.  However,  available  methods  will  be  those  that  are  supported  by 
empirical  and  theoretical  work  on  learning  and  transfer  of  training. 

For  training  primary  helicopter  tasks,  the  instructor  model  will  stimulate  a  flight 
simulation  capability  as  a  primary  mode  for  exercise  presentation.  However,  the  Instructor 
model  will  support  several  pedagogic  approaches,  including: 

■  Task  preview,  in  the  appropriate  modality  for  the  students  learning  style 

■  Step  and  procedure  cueing,  appropriate  to  the  student’s  learning  style 
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Immediate  corrective  feedback 


■  Task  performance  aids 

■  Probe  questions 

■  Positively  reinforcing  statements 

■  Task  cessation  and  restart  when  a  pre-set  error  limit  is  reached 

■  Changes  in  performance  requirements  as  the  student  becomes  more  proficient 

■  Part  to  whole  training 

■  Whole  to  part  training,  which  is  a  strategy  that  is  frequently  effective  with  visual-spatial  learners 

■  Discourse  referral,  in  which  the  on-line  instructor  refers  back  to  previous  guidance  or 
discussion 

These  approaches  can  be  described  and  illustrated  as  follows: 

Task  Preview:  Human  instructors,  when  interacting  with  novices,  frequently  describe 
the  task  that  is  to  be  performed  in  a  step-wise  fashion  to  allow  the  student  to  prepare  him/herself 
for  the  task,  and  to  guide  the  performance  of  the  task.  This  guidance  has  been  referred  to  as 
“advance  organizers”  in  the  literature  (Ausuble,  1960).  Advance  organizers  supply  the  student 
with  a  mechanism  by  which  he/she  can  begin  to  identify  the  relationships  among  new  knowledge 
and  skill  elements  with  those  that  have  been  previously  acquired.  This  mode  of  presentation  is 
appropriate  for  auditory  learners  (Silverman,  1995).  However,  for  those  students  whose 
preferred  learning  style  is  not  auditory,  other  modes  need  to  be  used  to  supplement  the  verbal 
task/step  description  to  supply  appropriate  organization  (Silverman,  1995).  Therefore  for  the 
adaptive  learning  system,  during  Task  Preview,  the  “instructor”  will  verbally  explain  the  current 
task  or  task  step  to  the  student.  For  students  who  are  auditory  learners,  the  information  will  be 
presented  auditorially.  For  those  students  who  are  visual-spatial  learners,  in  addition  to  the 
auditory  presentation,  a  visual  simulation  of  the  helicopter’s  flight  path  and  the  views  the  student 
will  have  will  be  presented  simultaneously  to  the  auditory  presentation.  For  kinesthetic  learners, 
the  system  will  request  them  to  place  their  hands  and  feet  on  the  controls.  The  system,  then 
while  presenting  the  auditory  preview,  will  move  the  controls  in  a  way  to  simulate  the  kinesthetic 
input  that  the  students  would  have  during  the  task  or  task  step. 

Cueing:  One  important  role  that  a  human  instructor  performs  for  a  novice  is  to  point 
significant  task  or  step  features  and  their  importance  is  task  performance  immediately  prior  to 
their  actual  use.  The  act  of  cueing  informs  the  student  of  when  a  signal  or  event  to  be  detected  is 
to  occur.  It  draws  the  student’s  attention  to  the  signal  so  that  the  student  may  analyze  it  and  act 
if  necessary  (Patrick,  1992).  Thus,  the  system  will  indicate  auditorially,  and  if  possible,  visually 
the  location  of  the  information  that  is  to  be  attended  to  or  the  control  that  is  to  be  manipulated. 
In  this  way,  the  information  will  be  made  available  to  the  student  model  as  it  is  needed  and 
therefore  will  be  more  likely  to  be  acted  upon. 
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Corrective  Feedback:  If  the  student  does  not  perform  a  maneuver  correctly,  then  the 
system  will  point  out,  based  on  the  assumed  mental  model  of  the  student  as  compared  to  the 
standard  for  performance,  where  the  error  has  been  made,  the  potential  reason  for  the  error,  and 
the  steps  to  correct  the  error.  This,  again,  is  a  method  frequently  used  by  human  instructors  to 
shape  student  behavior.  (Fox,  1993a;  Moore,  1993)  (See  Appendix  A) 

Performance  Aids:  Performance  aids  are  frequently  used  to  support  decision  making  in 
situations  in  which  decisions  are  based  on  complex,  dynamic  information  (Vazquez- Abad  & 
Winer,  1992).  Flying  a  helicopter  is  such  an  environment.  Helicopters  do  have  instrumentation 
that  serves  as  performance  aids.  However,  the  use  of  these  aids  may  be  too  difficult  for  a  novice 
to  initially  integrate  into  his/her  on-going  mental  model  to  support  performance.  Therefore,  we 
will  perform  research  to  determine  if  there  is  some  other  type  of  display  that  more  information  in 
a  way  that  is  more  useful  to  novices  as  they  learn  to  maneuver  the  helicopter. 

Probe  Questions:  One  way  that  human  tutors  gain  information  concerning  a  student’s 
thought  processes  is  to  ask  questions  (Fox,  1993b).  This  process  can  serve  multiple  functions. 
First,  the  answer  to  a  question  can  indicate  whether  or  not  a  trainee  has  available  in  working 
memory  a  piece  of  information.  If  the  student  does  not  have  that  information,  the  probe  question 
can  serve  as  cue  to  the  student  to  either  retrieve  the  information  or  if  it  a  question  focusing  on  an 
observable,  make  the  needed  observation.  Additionally,  the  way  in  which  a  student  answers  a 
question  can  indicate  changes  in  stress.  For  example,  if  the  student  answers  quickly  and  tersely, 
and  in  a  vocal  level  that  is  at  a  higher  decibel  and/or  frequency  level  than  is  normal,  then  it  is 
possible  that  the  student’s  stress  level  is  increasing.  As  stress  level  decreases,  there  is  likely  to 
be  a  reduction  in  amount  of  information  available  in  and  to  working  memory.  Additionally,  the 
student  will  begin  to  “tunnel’  both  perceptually  and  cognitively. 

Positive  Reinforcement:  It  is  a  well-established  principle  that  supplying  positive 
feedback  indicates  to  a  student  what  he/she  is  doing  correctly  within  a  task  Additionally, 
according  to  flight  instructors  interviewed  dining  the  development  of  this  description,  supplying 
positive  feedback  aids  in  fostering  a  high  level  of  self  esteem  in  the  student.  The  instructors  have 
noted  that  when  a  student  feels  as  though  he  or  she  is  accomplishing  tasks  current  satisfactorily, 
they  feel  more  confident  with  regard  to  subsequent  task  performance.  If  the  student  feels 
confident  in  their  performance,  they  are  less  likely  to  suffer  from  the  negative  impact  of  stress 
upon  cognition  and  behavior. 

Task  Restart:  If  a  student  errs  from  the  task  by  a  defined  amount,  the  system  will  require 
the  student  to  restart  the  task.  This  approach  will  help  to  ensure  that  students  do  not  learn 
incorrect  procedures.  If  students  learn  and  rehearse  for  any  length  of  time  incorrect  actions 
during  training,  there  is  a  chance  these  inappropriate  actions  may  transfer  to  the  actual  flight 
setting,  rather  than  the  correct  ones.  Use  of  the  these  inappropriate  behaviors  could  potentially 
have  catastrophic  results.  However,  it  should  be  pointed  out,  that  later  in  the  training,  it  may  be 
useful  to  train  students  in  ways  to  correct  error  situations. 

What  is  defined  as  unacceptable  will  vary,  given  the  student’s  experience  level  on  the 
task  or  on  similar  ones,  such  that  the  standards  will  be  more  lenient  initially,  and  will  become 
more  strict  as  the  student  gains  more  experience.  This  is  very  similar  to  the  way  flight 
instructors  shape  correct  performance  of  each  student.  The  system  will  supply  the  student  will 
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corrective  feedback  based  on  the  type  of  error  detected  and  the  assumed  mental  model  of  the 
student.  Positive  feedback  in  the  form  of  information  of  correctly  performed  task  steps  will  also 
be  supplied.  Once  feedback  has  been  supplied,  the  student  will  be  allowed  to  restart  the  exercise. 

Part  to  Whole  Training:  Many  training  and  instructional  systems  present  tasks  in  a  step¬ 
wise  fashion,  starting  with  teaching  each  step  or  part  of  the  task,  then  training  combined  sets  of 
steps,  and  then  finally  all  steps  together.  This  method  of  presentation  is  suitable  for  certain  types 
of  learners  (auditory,  sequential  learners)  or  certain  types  of  tasks,  especially  those  that  require 
one  to  follow  a  procedural  sequence  (Fisk,  1987;  Mane,  1984).  Many  helicopter  maneuver  tasks 
can  be  broken  down  into  their  constituent  elements,  and  that  is  what  flight  line  instructors  do  as 
they  train  the  tasks.  However,  there  may  be  certain  aspects  of  the  task  that  cannot  be  broken 
down  and  must  be  trained  as  a  whole,  but  within  a  simplified  situation.  It  is  expected  that 
students  who  are  interacting  with  the  system  early  in  the  training  cycle  will  require  more 
simplified  exercises  than  they  will  require  later  in  the  cycle. 

Whole  to  Part  Training:  Another  approach  to  the  presentation  of  tasks  to  be  performed 
is  to  present  the  task  as  a  whole,  and  let  the  student  attempt  to  perform  the  task.  This  approach 
often  is  used  in  situations  in  which  it  is  difficult  to  break  a  task  down  into  constituent  sub-tasks 
or  for  use  with  students  who  tend  to  see  task  elements  as  inter-related  (Wightman  &  Lintem, 
1985;  Silverman,  1995).  If  the  student  has  difficulties,  based  on  the  observation  of  performance 
and  then  the  assumption  of  the  current  mental  model,  the  system  will  focus  on  the  specific  task 
aspects  that  seem  to  be  the  locus  of  the  incorrect  performance.  It  should  be  pointed  out  that 
whole-to-part  learning  is  frequently  a  better  approach  for  visual-spatial  learners  than  is  a  more 
sequential  one  such  as  part-to-whole  training  tends  to  be  (Silverman,  1995).  Additionally,  since 
flying  a  helicopter  has  many  task  elements  that  must  be  performed  simultaneously,  a  student 
needs  to  have  an  understanding  of  those  task  elements  in  their  integrated  form  early  in  the 
learning  cycle  in  order  to  perform  in  an  acceptable  way  on  subsequent  tasks.  As  mentioned 
previously,  in  the  event  that  it  is  not  feasible  to  break  a  task  or  part  of  a  task  into  constituent 
elements,  the  system  will  present  a  simplified  version  of  the  integrated  task  or  step. 

Discourse  Referral:  Discourse  referral  is  a  critical  feature  of  human  discourse,  and  it  is 
widely  used  by  human  tutors  (Moore,  1993).  Human  tutors  frequently  refer  back  to  information 
previously  presented  to  students  in  order  to  bring  that  information  and  associated  information 
into  working  memory  of  the  student.  Such  a  strategy  will  be  implemented  within  the  Instructor 
of  the  adaptive  system.  In  cases  in  which  the  student  is  not  performing  as  expected  and  there  is  a 
record  of  previous  feedback  or  guidance  supplied  by  the  on-line  instructor,  the  instructor  will 
refer  back  verbally  to  the  previously  supplied  information.  The  reference  will  be  concise,  but 
detailed  enough  to  prompt  the  student  to  remember  the  previous  presentation. 

The  Instructor  Model  will  be  implemented  as  an  accessible  knowledge  base  to  hold 
content  and  pedagogic  items  keyed  by  items  that  reside  in  the  Student  Model.  These  items 
include: 


■  Student  learning  style, 

■  Student  expertise, 

■  Level  student  current  KSAs  (knowledge,  skills,  and  abilities), 
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■  Student  personality  attributes  and  attitude, 

■  Student  fatigue  and  stress  level, 

■  Student  demographics, 

■  Instructor  presentation  style, 

■  Exercise  type 

■  Interactions  of  the  above  factors  as  can  be  supported  by  current  theory  and  research  in 
psychology. 

These  items  are  more  fully  described  within  the  discussion  of  the  Student  Model. 

Student  Model 

This  aspect  of  the  system  will  support  two  primary  roles.  First,  it  will  include  the 
dynamic  model  of  the  student’s  understanding  of  the  task  elements  and  task  procedures,  as  the 
training  exercise  progresses.  It  will  also  contain  data  for  each  student  that  will  be  used  to  select 
and  modify  the  specific  exercises  and  feedback  styles  used  by  the  Instructor  Model.  The 
student’s  conceptual  model  must  include  the  mental  model  held  by  the  student  of  the  situation 
and  environment,  along  with  the  learning  styles  previous  training,  etc. 

In  addition  to  the  student  mental  model,  the  Student  Model  will  house  information  that 
will  constrain  the  form  and  content  of  the  student  model  and  will  inform  the  Instructor  Model  as 
to  the  types  of  feedback  and  interaction  styles  that  are  appropriate  to  each  specific  student.  This 
type  of  information  will  include  current  student:  learning  styles,  expertise  level,  KSAs 
(knowledge,  skills,  and  abilities),  personality  factors,  attitude,  fatigue  and  stress.  Each  of  these 
factors  is  described  below. 

Learning  Styles.  Research  suggests  that  there  are  individual  cognitive  processing 
differences  among  people  with  certain  variations  more  identifiable  than  others.  These  are 
referred  to  frequently  as  “learning  styles”  (Silverman,  1995).  Individuals  have  preferences  for 
mode  of  presentation  of  information  that  seems  to  be  tied  to  how  fast  they  can  process 
information  presented  within  different  modalities  (Just  &  Carpenter,  1992).  The  learning  styles 
that  the  system  will  support  will  include: 

■  Auditory  processor:  An  auditory  learner  needs  to  have  information  presented  step  by  step, 
sequentially,  via  the  auditory  mode.  This  means  that  the  Instructor  Model  must  have  tasks 
broken  into  steps  for  presentation  and  assessment 

■  Visual-spatial  processor:  A  visual-spatial  learner  frequently  thinks  pictorially  and  processes 
visual  material  more  quickly  that  auditory  material.  Task  needs  to  be  presented  initially  visually 
as  a  whole,  with  auditory  back  up.  Feedback  and  preview  steps  need  to  be  visually  presented. 

■  Kinesthetic  processor:  A  kinesthetic  learner  needs  to  have  device  controls  that  supply 
appropriate  feedback/resistance  as  he/she  learns  to  perform  the  task.  The  current  Intelligent 
Flight  Trainer  (IFT)  does  this  to  a  certain  extent. 
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The  system  will  need  to  have  knowledge  available  concerning  each  student’s  preferred 
method  of  learning.  To  ascertain  the  student’s  preference  beforehand,  the  student  will  enter 
answers  to  a  series  of  yes-no  questions  presented  on  the  exercise  control  (data  entry)  screen. 
These  questions  will  query  the  student  in  a  straightforward  way  as  to  how  he  or  she  has  best 
learned  different  types  of  material  and  tasks  in  the  past.  This  information  will  be  used  by  the 
Instructor  Model  to  select  the  best  methods  for  guiding  and  supporting  the  student. 

Student  expertise  level.  Within  the  range  of  students  in  the  course,  expertise  level 
consist  of  an  indication  of  where  within  the  training  program  the  student  is  and  the  training 
objectives  he  or  she  has  already  met.  This  is  information  that  will  be  entered  by  training 
personnel  via  the  training  manager  or  recorded  from  previous  automatic  instruction. 

Student  KSAs.  The  student’s  KSAs  actually  are  incorporated  in  multiple  aspects  of  the 
Student  Model  and  thus  are  described  elsewhere.  However,  for  each  student  the  KSAs  will  be: 

■  Knowledge  items  as  reflected  by  the  student  mental  model 

■  Skills  as  reflected  by  current  performance  data 

■  Abilities  as  reflected  in  performance  on  components  of  selection  tests 

Student  Personality  Factors.  Aspects  of  a  student’s  personality  will  in  all  likelihood 
impact  their  ability  to  learn  the  presented  tasks  (Tannenbaum,  et.  al.,  1991).  If  the  system  can 
take  such  personality  factors  into  account  during  presentation  and  remediation  and  modify  its 
response  in  relation  to  those  factors,  it  may  lead  to  more  effective  training.  Including  this 
capability  in  the  system  will  allow  researchers  to  investigate  the  impact  of  personality  variables 
on  learning.  Thus  the  initial  set  of  variables  could  include: 

■  Extravert/introvert 

■  Thoughtful/Impulsive 

■  Analytic/Intuitive 

Information  concerning  these  variables  would  be  derived  from  student  performance  on 
standardized  personality  tests  Training  staff  or  researchers  would  enter  these  data  via  the 
Training  Manager  function. 

Student  Attitude.  According  to  experienced  instructor  pilots,  the  attitude  of  the  student 
can  be  an  important  factor  in  how  quickly  helicopter  flight  tasks  can  be  learned.  This  finding  is 
supported  in  the  literature  (see  for  example,  Noe,  1986).  Attitude  can  be  potentially  broken  into 
multiple  factors,  each  of  which  could  have  some  impact  on  training.  The  components  that  could 
impact  learning  include: 

■  Passive  and  not  taking  initiative  >  Aggressive  and  overly-confident 

■  Responsive  to  direction  <— >  Recalcitrant  to  direction 
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Information  concerning  a  student’s  attitude  would  be  based  on  instructor  observation  and 
entered  into  the  system  via  the  Training  Manager. 

Student  Fatigue.  Physical  fatigue  and  sleep  deprivation  can  have  a  negative  impact  on 
cognitive  functioning  (Orasansu  &  Backer,  1996).  Therefore  the  system  will  ascertain  the 
mental  fatigue  level  of  the  student  before  he  or  she  begins  an  exercise.  The  system  will  then 
adjust  the  problem  difficulty  to  allow  for  the  expectation  of  potential  cognitive  overload.  To 
determine  the  level  of  fatigue,  the  student  will  be  queried  as  to  the  number  of  hours  of  sleep  the 
student  had  the  previous  night  before  he  or  she  begins  the  exercise. 

Psychological  Stress.  Psychological  stress,  resulting  from  various  factors,  has  been 
shown  to  have  a  detrimental  impact  on  cognitive  processing,  once  it  reaches  a  non-optimal  level 
of  arousal  (Wickens,  et  al.,  1991;  Stokes  &  Kite,  1994;  Hockey,  1986).  The  major  negative 
impact  is  “  “tunneling.”  Tunneling  affects  perception,  cognition  and  decision  making  (Wickens, 
et  al,  1991).  Its  impact  is  that  an  individual  under  stress  will  attend  to  fewer  aspects  of  his  or  her 
environment,  resulting  in  a  more  restricted  awareness  of  the  current  situation.  Additionally,  an 
individual  under  stress  will  be  less  likely  to  arrive  at  novel  or  rarely  used  interpretations  in  the 
current  situation.  Finally,  stress  can  cause  the  individual  to  focus  on  a  limited  number  of  task 
items.  In  other  words  the  working  memory  is  degraded  under  stressful  conditions.  According  to 
instructor  pilots,  when  students  are  initially  learning  helicopter  flight  tasks,  they  become 
overwhelmed  by  the  task  and  their  levels  of  arousal  and  stress  increase.  This  results  in  a 
lessening  of  situation  awareness  and  an  over-focus  on  the  current  guidance  supplied  by  the 
instructor.  A  final  impact  of  psychological  stress  is  to  cause  a  strategic  shift  in  which  speed  is 
accentuated  over  accuracy  of  performance  (Hockey,  1986). 

The  adaptive  learning  system  will  gather  and  track  data  concerning  the  student’s  level  of 
stress.  Stress  levels  will  be  assessed  by  capturing  speech  input  at  the  beginning  of  the  task,  and 
at  points  during  the  task.  Stress  indicators  such  as  changes  in  speech  rate  and  decibel  and 
frequency  levels  will  be  calculated.  Once  a  specified  level  of  change  has  occurred,  thus 
indicating  that  the  student  has  potentially  reached  point  of  stress  that  is  detrimental  to  the 
continuation  of  the  task,  the  system  will  stop  the  current  exercise,  supply  feedback  to  the  student 
appropriate  to  the  student’s  current  performance  and  assumed  mental  model.  Then  the  system 
will  present  a  somewhat  easier  scenario  to  the  student. 

Training  Analyst 

The  training  analyst  function  collects  student  performance  data  so  that  the  Student  Model 
can  be  updated  to  reflect  the  most  current  high-probability  model  of  the  student.  Collected  data 
will  include  both  physical  and  verbal  responses  by  the  student  to  the  current  task  requirements 
and  external  information.  The  Training  Analyst  will  contain  performance  indicators  that  will 
show  when  the  student  should  restart  the  exercise.  There  will  be  multiple  indicators  for  any 
particular  exercise,  with  the  specific  indicator  to  be  selected  based  on  the  student’s  current 
expertise  level,  at  a  minimum. 

The  Training  Analyst  will  also  contain  algorithms  for  computing  performance  statistics 
within  an  individual’s  performance  and  across  individuals,  and  tied  to  any  specified  training 
component  or  pedagogic  technique.  This  will  permit  the  training  analyst  to  provide  feedback  to 
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the  Instructor  Model.  Time  spent  on  each  training  element  necessary  to  meet  a  training  objective 
can  also  be  calculated  in  order  to  assess  the  training  efficiency  of  the  system  as  it  relates  to  the 
effectiveness  of  the  various  included  pedagogic  methodologies 

Training  Manager 

The  Training  Management  Function  will  allow  human  input  of  student  background 
information,  scenario  elements,  and  training  objectives  for  program,  and  will  present  results  of 
analyses  to  training  personnel  and  researchers.  Given  that  the  proposed  system  is  both  a  training 
and  a  research  tool,  it  is  important  to  allow  appropriate  personnel  access  to  databases  internal  to 
the  system  for  updating  and  modification.  Scenario  generation  is  closely  tied  to  the  host 
simulator  and  the  facilities  will  have  to  be  analyzed  to  determine  the  required  modifications  or 
enhancements. 

Presentation  Manager 

The  Presentation  Manager  will  receive  input  from  the  Instructor  and  Student  Models  and 
content  information  to  select  and  present  the  material  most  appropriate  to  the  current  student  in 
the  most  appropriate  format  and  style.  There  are  certain  established  parameters  to  the 
Presentation  Manager,  however.  The  overall  system  is  scenario  and  simulator-based,  with 
supplemental  guidance  and  instruction.  Therefore  the  presentation  must  stay  within  the 
constraints  established  by  such  a  system.  The  visual  elements  of  the  presentation  will  contain 
simulated  external  and  internal  environments  need  to  fly  a  given  helicopter  for  the  selected 
training  maneuvers.  Additional  visual  presentations  may  include  displays  that  overlay  the 
primary  displays  in  order  to  supply  cueing  information  to  the  student.  There  will  also  be  an 
auditory  component  to  the  presentation.  The  system  will  speak  to  the  student  as  described 
elsewhere,  supplying  task  guidance,  cueing,  and  feedback.  The  system  will  also  ask  probe 
questions  as  a  data  gathering  mechanism,  the  results  of  which  will  be  input  into  the  Training 
Analyst  and  the  Instructor  Model. 
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Section 

3 


Summary 

Final  Summation  and  Key  Comments  on  the  Design 


Summary 


The  purpose  of  this  study  was  to  develop  an  approach  to  a  versatile  adaptive  system  with 
growth  potential  to  a  learning  instructional  system.  This  study  has  developed  a  model  set 
illustrating  how  a  single  knowledge  representation  technique  can  be  used  to  capture  mental 
models,  actual  situations  and  actual  behavior.  The  common  representation  of  conceptual  graphs 
gives  us  enormous  power  and  potential  to  manipulate,  analyze  and  exchange  all  these  kinds  of 
knowledge  within  one  framework.  We  have  introduced  an  approach  to  determining  how  correct 
behavior  can  be  distinguished  from  incorrect  behavior,  and  how  feedback  on  the  incorrect 
behavior  can  be  used  to  trigger  further  adaptation  in  the  instruction  sequence. 
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Appendix 

A 


Conceptual  Model  Demo 

nstration 

Task  Notes 

|  Conceptual  Modeling  | 

Potential  Training  Task  for  Student 

Current  Situation  at  Start  of  Task 

AC  in  air  at  elevation  of  45’,  air  speed  of  90  knots/hr,  movement  in  forward  direction,  no  wind;  there 
is  a  hangar  1200’  south  from  the  staging  area  and  on  the  right;  there  is  a  water  tower  west  of  the 
hangar. 

Task 

1)  Fly  south  from  staging  area  1200’. 

2)  Turn  right  (West)  90  degrees  when  parallel  to  the  hangar. 

3)  Change  elevation  to  70’ 

4)  Change  air  speed  to  1 20  knots/hour. 

5)  Fly  over  the  hangar  toward  the  water  tower. 

There  are  several  reasons  for  suggesting  this  task.  First,  it  requires  the  student  to  use  external 
visual  cues  that  are  somewhat  distant  while  performing  the  task.  We  can  discuss  in  our 
presentation  ways  to  track  eye  movements  to  determine  if  the  student  monitors  these  cues  or  not. 
If  not,  we  can  assume  that  for  our  dynamic  cognitive  model  of  the  student  the  relative  location 
information  supplied  by  the  cues  is  not  available.  Second,  student  pilots  often  don’t  realize  when  to 
use  the  pedals  for  turning  versus  the  cyclic,  (at  low  elevations  the  pedals  are  used).  If  we  have  our 
hypothetical  student  have  this  as  part  of  the  task  will  allow  us  to  demonstrate  an  observable  that 
would  indicate  an  aspect  of  the  student’s  mental  model.  Third,  since  student  pilots  initially  do  not 
understand  the  relationship  of  the  collective  and  cyclic  with  regard  to  pitch  control,  having  the 
student  change  air  speed  will  require  him/her  to  integrate  the  use  of  both  controls  to  ensure  level 
flight.  If  we  observe  tipping  forward,  we  can  assume  that  the  student’s  mental  model  does  not 
have  a  correct  representation  of  the  collective-cyclic  relationship.  We  can  then  posit  potential 
representations  that  the  student  does  have,  based  on  a  Bayesian  analysis  of  his/her  performance 
on  other  parts  of  this  task  or  on  subsequent  tasks.  Potential  representations  could  include:  1)  the 
student  thinks  that  the  collective  has  throttle  function  only  (very  common  representation  in  novices), 
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or  2)  the  student  underestimates  impact  of  the  collective  on  the  angle  of  attack  of  the  rotor  (a  more 
sophisticated  representation). 


Student  Profile 

For  our  example  student  (first  5  items  need  to  be  pre-loaded  into  trainee  database) 

■  Experience  level:  has  never  been  to  flight  line,  has  had  three  days  of  core  curriculum 

■  Demographics:  24  year  old  male  from  Pittsburgh,  PA 

■  Cognitive  strengths  (from  screening  tests):  strongly  visual-spatial 

■  Cognitive  weaknesses  (from  screening  tests):  low  on  auditory  processing 

■  Attitude  (from  personality  screening  tests/instructor  assessment):  Over-reactive/nervous 

■  Fatigue  level:  low 


Student  Knowledge  when  starting  the  exercise  (this  information  can  either  be  assumed  or  be 
determined  via  on-line  testing;  either  way,  we  should  assign  a  probability  value  of  having  these 
pieces  of  information  available  to  working  memory  at  the  start  of  the  exercise): 

■  Control  names 

■  Control  functions,  in  general 

■  Instrument  names  and  functions 

■  Distance  from  staging  area  to  hangar 

The  following  groupings  represent  the  student’s  potential  knowledge  state  as  the  exercise  begins 
(note  that  this  information  is  not  in  conceptual  graph  format,  but  I  think  the  information  to  begin 
building  such  representations  is  there).  However,  since  the  student  is  very  inexperienced,  some  of 
this  information  may  not  be  in  working  memory  at  the  start  of  the  exercise.  It  should  be  retrievable 
from  long  term  memory  or  easily  updated  if  the  planned  actions  are  followed.  Each  item’s 
availability  should  have  an  associated  probability  of  being  in  working  memory  and  a  probability  of 
being  available  to  working  memory  (e.g.,  exists  in  long  term  memory).  For  any  specific  student 
model,  these  probabilities  will  be  varied  based  on  assessed  background  knowledge  and 
personality  factors.  Later  in  the  training  exercise,  the  probabilities  will  be  modified  based  on  past 
experiences  within  the  current  simulation  situation  or  previous  simulator  sessions.  (In  other  words, 
there  will  be  a  cumulative,  dynamic  student  model  maintained  by  the  system  for  each  student  who 
uses  it.) 

CONTROLS  KNOWLEDGE 

1)  Collective:  function  is  like  throttle;  it  increases  air  speed 

2)  Cyclic:  functions  to  control  pitch 

3)  Pedals,  functions  to  off-set  torque 
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INSTRUMENT  FUNCTIONS  AND  CURRENT  STATES 


1)  Air  speed  indicator:  indicates  speed,  currently  at  90  knots/hr 

2)  Turn  &  Slip  indicator:  indicates  relative  positions  of  pedals,  currently  right  pedal  depressed 
enough  to  prevent  torque 

3)  Vertical  speed  indicator:  indicates  how  fast  AC  is  rising,  currently  0 

4)  Attitude  indicator:  indicates  AC  attitude  with  respect  to  the  level,  currently  AC  is  level 

5)  (and  with  some  AC — altimeter:  indicates  height  above  sea  level) 

STATE  OF  EXTERNAL  CUES 

1)  Current  AC  position  with  respect  to:  front  far  view  scanning  ahead:  runway,  front  near  view: 
staging  area,  right  far  view:  right  close  view:  staging  area  lines 

2)  Position  changes  based  on:  texture  gradient  change,  enlargement  of  selected  location  cueing 
objects,  speed  of  closure 

CURRENT  ACTIONS 

1)  Maintaining  positions  of  controls 

2)  Monitoring  far  cues 

3)  Monitoring  near  cues 

4)  Attending  to  training  direction 

PLANNED  ACTIONS  (this  is  the  amount  of  direction  that  the  system  gives  the  student  initially) 

1 )  Fly  straight  at  current  elevation 

2)  Fly  straight  south  for  1 200’ 

3)  Look  for  hangar 

KINESTHETIC  CONTROL  RESPONSE  EXPECTATIONS 
1 )  <no  current  expectations> 

Other  Impacts  to  the  Model  Contents 

There  are  other  aspects  of  the  student  pertaining  to  the  student  model  that  aren’t  directly 
components  of  the  model,  but  do  impact  the  probabilities  of  informational  or  task  planning  and  task 
action  components  entering  or  leaving  the  model  while  the  student  is  performing  the  task.  These 
are  some  of  the  student’s  personality  characteristics,  sociological  background,  and  cognitive 
processing  strengths  and  weaknesses.  The  characteristics  that  will  impact  the  probabilities,  and 
their  impact,  are: 
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Experience — As  experience  increases  the  likelihood  that  a  new  piece  of  information  related  to 
previously  learned  knowledge  will  be  integrated  will  increase.  Also,  as  experience  increases 
greater  amounts  of  information  related  to  previously  understood  information  will  be  learned  more 
quickly. 

Fatigue  level— As  fatigue  level  goes  up,  the  probability  of  new  information  being  learned  or 
previously  learned  information  being  available  decreases.  Additionally,  the  likelihood  that  certain 
tasks  such  as  monitoring,  will  be  correctly  performed  decreases.  The  likelihood  that  only  near  cues 
or  centered  cues  will  be  monitored  over  other  cues  will  increase. 

Affect  type— As  the  anxiety  level  of  the  student  increases,  the  probability  of  new  information  or 
plans  entering  the  mental  model  decreases.  The  probability  of  monitoring  only  close  informational 
cues  such  as  instruments  or  close  visuals  will  increase.  The  probability  of  over-focus  on  the  most 
recent  correction  by  the  instructor  will  increase. 

Learning  style/cognitive  strengths— Visual  learners  will  have  a  higher  probability  of  acquiring 
visually  available  information  or  retrieving  from  long  term  memory  that  has  been  presented  visually, 
than  of  receiving  or  retrieving  auditory-based  or  kinesthetically-based  processed  information.  The 
same  for  all  other  preferred  learning  styles,  in  that  they  have  a  higher  probability  of  receipt  and  use 
of  information  within  their  preferred  modality. 

Attitude — The  student’s  attitude  can  negatively  or  positively  impact  his/her  ability  to  learn 
information  that  is  presented.  A  negative  attitude  will  lessen  the  likelihood  that  information  will  be 
acquired  and  used.  A  positive  attitude  may  increase  the  probability  of  an  element  entering  the 
student’s  mental  model. 

Impacting  Simulation  Factors 

There  will  be  certain  aspects  of  the  simulated  flight  situation  that  will  impact  the  student  model,  and 
will  interact  with  the  above  factors,  include: 

Total  amount  of  information  to  be  processed  during  any  instance  of  the  simulation  and  the 
task  to  be  performed — As  the  number  of  informational  items  to  be  attended  to  and  task  steps  to 
be  performed  or  planned  for  simultaneously  increases,  the  probability  of  any  specific  plan  or 
information  item  be  attended  to  will  decrease.  As  expertise  increases,  the  probability  of  managing 
multiple  simultaneous  information  items  and  plans  to  be  executed  will  increase. 

Total  amount  of  information  coming  to  any  one  sensory  channel — As  the  amount  of 
information  to  a  specific  channel  increases,  so  does  the  processing  load  for  the  channel.  As  load 
increases,  the  likelihood  that  any  specific  information  in  the  channel  will  be  received,  or  if  received, 
attended  to  or  used,  decreases.  If  redundant  information  comes  via  multiple  channels,  the 
probability  of  information  receipt  and/or  use  increases. 

Note  that  to  a  large  extent  the  simulation-based  factors  can  be  controlled  by  the  instructor  model 
used  to  select  tasks  for  the  student. 

Impacting  Pedagogic  Factors 

The  use  of  specific  pedagogic  methods  will  increase  or  decrease  the  probabilities  associated  with 
whether  or  not  a  particular  student  given  a  specific  training  situation  will  modify  his/her  mental 
model  and  responses  in  a  certain  way.  These  factors  include: 
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Specific  task  selection — To  ensure  that  the  student’s  mental  model  will  have  the  greatest 
probability  of  changing  correctly,  the  instructor  model  will  need  to  select  tasks  based  on  an 
assessment  of  the  current  student  model  as  compared  to  overall  learning  goals  that  have  been 
established  for  this  level  of  the  curriculum.  For  example,  if  the  student  does  not  understand  the 
pitch  control  aspect  of  the  collective,  then  the  task  should  require  the  student  to  practice  using  the 
collective  and  the  cyclic  in  concert  to  control  the  pitch.  Task  selection  should  also  contain 
identifiable  points  for  gathering  observable  data  in  order  to  update  the  student  mental  model  or 
potential  models. 

Rate  of  task  content  presentation/Task  step  previewing — To  increase  the  probability  of  correct 
performance,  as  a  default  presentation  method  for  most  students,  the  instructor  model  should  first 
describe  the  whole  task,  then  as  each  step  is  to  be  performed.  The  “instructor”  should  state  the 
step.  The  detail  level  of  the  step  should  be  finer  grained  for  more  novice  students.  This  will  mean 
that  very  novice  students  will  be  receiving  constant  task  guidance.  Students  who  have  shown 
proficiency  on  specific  aspects  of  the  tasks  should  receive  less-detailed  preview. 

Style  of  verbal  presentation— To  increase  the  probability  of  information  being  added  to  the 
student’s  mental  model,  the  verbal  presentation  style  of  the  “tutor”  should  be  compatible  with 
certain  aspects  of  the  student.  For  example,  if  the  student  is  from  the  South,  speech  presentation 
should  be  slower  than  for  northern  students.  Also,  more  highly  anxious  students,  as  reflected  on 
personality  portions  of  screening  tests,  should  receive  more  detailed  task  guidance  and  more 
frequent  positive  feedback  than  the  average  student.  Belligerent/arrogant  students  should  receive 
terse  guidance. 

Task  aiding — If  the  pedagogic  module  supplies  task  aiding  such  as  highlighting  important  sources 
of  information,  and  does  so  in  a  way  that  makes  use  of  the  student’s  learning  style,  the  probability 
of  acquiring  the  information  will  increase. 

Feedback  style— The  style  and  frequency  in  which  the  pedagogic  component  supplies  feedback 
should  be  based  on  student  attitude  and  learning  style.  Content  should  be  based  on  student 
performance  or  assumed  most  probable  current  mental  model. 

Example  Operation 

Student  Mental  Model  During  Step  1  of  Task 

The  student  begins  the  task  with  the  previously  described  mental  model. 

The  instructor  model  says:  “Fly  south  1200’.  Maintain  elevation  and  speed” 

Student  looks  at  his  controls  (observed  by  system  via  gaze  or  head  movement  detector). 

Student  looks  out  window  to  distant  end  of  runway  (observed  via  gaze  or  head  movement 
detector). 

The  student  may  demonstrate  certain  behaviors  at  this  point.  He/she  may: 

Maintain  all  controls  in  the  same  positions  as  they  were  placed  at  the  beginning  of  the  exercise, 
thus  maintaining  course,  speed  and  elevation. 

Move  one  or  more  of  the  controls  (data  captured  by  the  system,  either  based  on  detection  of 
control  changes  or  AC  changes). 
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If  the  situation  is  the  first  case,  then  we  can  assume  that  the  section  of  the  student’s  mental  model 
concerning  the  kinesthetic  response  expectations  of  the  controls  has  been  immediately  filled 
correctly,  at  least  temporarily. 

If  the  student  doesn’t  not  maintain  all  controls  in  their  initial  positions,  then  kinesthetic  response 
expectations  of  the  controls  portion  of  the  student  model  will  be  updated,  based  on  the  control  that 
has  been  moved.  Given  the  size  of  impact  on  the  AC  of  movement  of  any  control,  the  tolerance 
level  should  be  very  low  before  the  model  is  updated  and  the  Instructor  model  offers  feedback. 
Actually,  one  can  expect  that  a  beginning  pilot  will  have  difficulty  maintaining  the  controls  in  their 
original  position,  so  each  time  the  controls  go  out  of  position,  the  Instructor  model  should  indicate 
immediately  which  control  needs  adjustment,  and  subsequently  which  other  controls  need  further 
position  modification.  (Please  note  that  our  hypothetical  student  will  also  require  frequent  positive 
feedback  statements  as  his  model  changes  to  reflect  learning  of  the  task.) 

Once  the  student  has  flown  most  of  the  1200’,  we  assume  the  cognitive  model  at  this  point  is: 

CONTROLS  KNOWLEDGE 

1)  Collective:  function  is  like  throttle;  it  increases  air  speed 

2)  Cyclic:  functions  to  control  pitch 

3)  Pedals,  functions  to  off-set  torque 
INSTRUMENT  FUNCTIONS  AND  CURRENT  STATES 

1)  Air  speed  indicator:  indicates  speed,  currently  at  90  knots/hr 

2)  Turn  &  Slip  indicator:  indicates  relative  positions  of  pedals,  currently  right  pedal  depressed 
enough  to  prevent  torque 

3)  Vertical  speed  indicator:  indicates  how  fast  AC  is  rising,  currently  0 

4)  Attitude  indicator:  indicates  AC  attitude  with  respect  to  the  level,  currently  AC  is  level 

5)  (and  with  some  AC — altimeter:  indicates  height  above  sea  level) 

STATE  OF  EXTERNAL  CUES 

1)  Current  AC  position  with  respect  to:  front  far  view  scanning  ahead:  runway,  front  near  view 

2)  Position  changes:  far  end  of  runway  approaching  (speed  of  closure) 

CURRENT  ACTIONS 

1 )  Maintaining  positions  of  controls 

2)  Monitoring  far  cues — end  of  runway 

3)  Monitoring  near  cues — staging  area 

4)  Attending  to  training  direction 
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PLANNED  ACTIONS  (this  is  the  amount  of  direction  that  the  system  gives  the  student  initially) 

1)  Fly  straight  at  current  elevation 

2)  Fly  straight  south  for  1 200’ 

KINESTHETIC  CONTROL  RESPONSE  EXPECTATIONS 

1)  <contents  based  on  student  performance  during  previous  portion  of  task,  but  for  each  control, 
it  will  be  in  the  form  of  “X  size  change  of  control  equals  X  size  change  of  AC  performance’^ 

At  this  point  the  student  receives  new  input  from  the  instructor  model.  ‘Turn  due  west  at  the 
hangar.”  There  are  several  behavioral  possibilities  after  this  direction.  The  student  may: 

■  Fly  past  the  hangar  and  never  turns. 

■  Turn  west,  but  not  due  west. 

■  Use  the  cyclic  to  make  turn  rather  than  the  right  peddle. 

■  Both  2  and  3 

■  Turned  at  the  proper  location,  but  either  late  or  early 

■  Crash  into  the  ground 

■  Something  else 

Each  of  these  behavioral  possibilities  can  reflect  multiple  possible  mental  models.  For  example,  if 
the  student  flies  past  the  hangar,  he/she  could  have: 

Not  monitored  out  the  front  and  right  windows  for  the  hangar.  This  possibility  may  be  supported  or 
ruled  out  by  eye  or  gaze  tracking,  or 

Not  remembered  how  or  which  controls  to  use  to  make  a  turn.  This  possibility  may  be  supported 
by  data  gathered  from  control  actuation. 

If  the  student’s  mental  model  indicated  that  the  student  had  not  attended  to  external  cues,  the 
system  would  stop  the  scenario,  and  point  out  the  needed  cue  both  verbally  and  visually.  The 
visual  indicator  would  be  a  box  surrounding  the  cue  of  importance.  After  the  intervention,  the 
training  session  would  restart. 

In  the  case  of  second  behavioral  possibility,  “Turn  west,  but  not  due  west,”  the  following  are 
possible  reasons  for  the  error  in  performance.  The  student: 

■  Did  not  understand  the  concept  of  “due  west,”  or 

■  Over  or  under  used  controls  while  making  turn.  This  possibility  may  be  supported  or  ruled  out 
by  data  gathered  form  control  actuation. 

If  the  system  determined  that  the  student  did  not  understand  the  concept  of  “due  west,”  the 
simulation  would  be  stopped  and  the  term  explained.  If  the  system  determined  over  or  under  use 
of  a  control,  the  system  would  stop  the  simulation  and  would  stress  the  need  to  make  small 
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changes,  possibly  with  an  exercise  on  control  moderation.  The  style  of  the  feedback  would  match 
the  student’s  learning  style.  After  the  intervention,  the  training  session  would  restart. 

For  the  third  behavioral  possibility,  “cyclic  used  for  turn,  not  peddle,”  control  use  data  would  indicate 
if  this  were  the  case.  If  so,  the  mental  model  might  reflect  one  of  four  possibilities.  The  student 
might  think: 

1)  The  cyclic  is  always  used  for  turning. 

2)  The  cyclic  is  used  for  turning  for  any  elevation  over  X  number  of  feet  (with  X  being  less  than 
the  current  elevation  of  the  AC). 

3)  The  pedals  are  used  only  for  anti-torquing. 

Both  1  and  3 

However,  determination  of  the  exact  student  model  for  this  type  of  error  is  not  really  necessary. 
Another  option  is  to  have  the  instructor  module  not  gather  more  information  at  this  time,  but  to 
explain  the  rule  concerning  pedal  use  for  turning  at  elevations  lower  than  50’.  The  student  model 
would  be  updated  to  reflect  a  correct  understanding  of  the  relationship,  after  the  intervention.  The 
student  would  begin  the  simulation  again. 

In  the  event  that  during  a  run  of  the  simulation  subsequent  to  the  first  instructor  intervention,  the 
student  errs  on  this  step  of  the  task,  the  instructor  model  should  collect  more  data  to  determine  the 
exact  problem  with  the  student  model.  This  can  be  done  in  either  of  two  ways.  The  instructor 
model  could  present  the  student  with  either:  1)  two  multiple  choice  questions  about  cyclic  and 
peddle  use  (this  means  having  pre-developed  questions),  or  2)  a  new  simulation  task  in  which  the 
student  must  make  a  turn  at  an  even  lower. 


Example  Adaptive  Instruction 


This  section  gives  a  brief  example  showing  how  conceptual  graphs  can  represent  the  knowledge 
required  to  perform  adaptive  instruction  within  the  simulated  training  environment.  We  will  show  a 
simple  two-step  task,  where  the  student  performs  the  first  step  correctly,  then  the  second  step 
incorrectly.  The  terms  expected  and  assumed  mean  things  that  are  in  the  system  preflight.  The 
terms  observed  and  actual  are  things  that  occur  during  the  simulation. 
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Figure  A-1  Waypoint  sequences 


The  process  consists  of  a  sequence  of  conceptual  waypoints,  where  each  waypoint  represents  a 
step  in  the  instructional  process.  Linking  these  waypoints  together,  we  can  establish  a  single 
"lesson"  of  instruction. 

Example  Task 

You  are  heading  south  at  45  ft  altitude.  When  you  reach  the  hangar,  turn  right  and  ascend 
to  120  ft. 

We  will  present  two  conceptual  waypoints  in  this  task.  The  waypoints  correspond  roughly  to  the 
stages  of  the  task:  turn  right,  and  ascend. 

Waypoint  A 

For  simplicity,  we  propose  a  session  where  the  aircraft  is  already  moving.  Future  work  will  address 
the  issues  of  the  student's  maintaining  a  set  course  and  speed.  The  waypoint  is  the  point  at  which 
the  instructor  unit  issues  a  directive. 

Current  situation  -  Waypoint  A 

The  current  situation  at  the  start  of  the  training  session  is  pre-defined  as  shown  in  Figure  1.  The 
aircraft's  movement  is  a  velocity  whose  heading  is  180  degrees  and  whose  airspeed  is  70  knots. 
The  aircraft's  location  is  a  place  with  height  45  ft  and  coordinates.  There  is  a  hangar  at  a  place  h 
and  a  water  tower  at  a  place  w. 


Note  that  the  student  should  think  (among  other  things)  that  the  cyclic  controls  the  pitch,  and  the 
collective  controls  the  altitude  and  velocity. 


Figure  A-3  Planned  instructor's  direction  to  student  -  waypoint  A 

The  instructor  unit  gives  the  student  a  direction:  Cause  the  aircraft  to  have  a  direction  of  west  at 
the  hangar.  This  graph  is  somewhat  simplified  for  the  purposes  of  this  illustration. 


A-10 


Figure  A4  Expected  student  behavior  -  waypoint  A 
The  expected  student  behavior  is  to  move  the  cyclic  to  the  right. 


Figure  A-5  Expected  resulting  situation  -  waypoint  A 


If  the  student  performs  the  correct  behavior,  the  aircraft  will  now  be  headed  west,  with  all  other 
attributes  the  same. 


Actual  Situation: 
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Figure  A-6  Observed  student  behavior  -  waypoint  A 

In  this  environment,  we  have  access  to  the  details  of  a  student's  actions.  We  suppose  in  this 
example  the  following  observations: 

■  Student  moved  the  cyclic  to  the  right 


Aril 


■  Student  moved  a  pedal. 


Actors  linked  to  the  simulator  will  allow  us  to  determine  what  controls  were  activated,  resulting  in 
the  graph  below. 

For  future  work,  we  will  be  able  to  observe  other  behaviors.  For  this  example,  let  us  suppose  that 
we  also  observe  the  following: 

■  Student  makes  eye  contact  with  the  compass  for  500  msec 

■  Student  makes  eye  contact  with  the  airspeed  indicator  for  300  msec. 

Elements  of  cognitive  theory  allow  us  to  assume  that  these  readings  have  now  entered  the 
student's  working  memory.  We  intend  to  model  these  observations  in  the  future. 


After  performing  the  instructor's  directive,  the  aircraft  should  be  heading  west  and  located  at  the 
hangar. 


Expected  Situation: 


Figure  A-8  Inferred  mental  model  -  waypoint  A 


Although  the  purpose  and  actions  of  the  collective  may  be  in  the  student's  actual  mental  model,  we 
have  no  means  of  directly  knowing  that,  since  the  student  did  not  activate  the  collective  during  this 
observation.  We  therefore  infer  that  the  student  knows  the  cyclic  controls  the  heading  and  pitch 
and  the  pedal  controls  the  heading. 
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■  Compare  the  actual  (simulated)  situation  with  the  expected  situation. 

■  Compare  the  student's  observed  behavior  with  the  expected  behavior. 

■  Compare  the  student's  inferred  mental  model  with  the  expected  mental  model. 

In  waypoint  A  of  this  example, 

■  the  actual  situation  matches  the  expected  situation, 

■  the  observed  behavior  includes  the  expected  behavior  (without  conflicts)  and 

■  the  student's  inferred  mental  model  is  included  in  the  expected  mental  model  (i.e.,  we  have  not 
inferred  anything  that  should  not  be  in  the  expected  model). 

We  can  therefore  reasonably  conclude  that  the  task  was  performed  correctly  and  that  the  mental 
model  is  compatible  with  what  we  expect  it  to  be.  No  adaptation  is  required. 
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Waypoint  B 


We  now  consider  the  second  waypoint,  after  the  student  has  successfully  turned  to  the  west.  The 
instructor  directs  the  student  to  ascend  to  120ft.  (We  assume  the  intent  is  to  maintain  course  and 
speed.) 

Current  situation  -  Waypoint  B 

Note  that  the  actual  situation  reflects  that  the  aircraft  is  now  heading  west  and  is  located  at  the 
hangar. 
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Figure  A-10  Assumed  mental  model  -  waypoint  B 


Same  as  for  Waypoint  A,  since  we’ve  detected  no  differences  in  expected  behavior.  In  future  work, 
we  will  be  able  to  increase  the  confidence  level  for  the  relations  involving  the  cyclic  and  pedal, 
since  the  student  did  operate  those  controls  correctly. 


Assumed  Mental  Model: 
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Figure  A-1 1  Planned  instructor's  direction  to  student  -  waypoint  B 
The  instructor  now  directs  the  student  to  cause  the  aircraft  to  have  an  altitude  of  120ft. 


A-1 4 


Figure  A-12  Expected  student  behavior  -  waypoint  B 


In  response  to  the  instructor’s  directive,  the  student  is  expected  to  lift  and  rotate  the  collective,  to 
gain  altitude  without  losing  speed. 


Expected  Student  Behavior: 
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Figure  A-13  Expected  resulting  situation  -  waypoint  B 

After  performing  this  directive,  the  aircraft  should  be  heading  due  west  at  an  altitude  of  120ft. 
Expected  Situation: 
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Figure  A-14  Observed  student  behavior  -  waypoint  B 
We  make  the  following  observations  of  the  student's  behavior. 

■  The  student  moved  the  cyclic  back. 
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■  The  student  observed  the  altimeter  for  1 200  msec. 


Since  the  student  was  expected  to  use  the  collective,  this  should  lead  to  our  system  detecting  a 
problem.  We'll  show  that  below. 


Figure  A-15  Inferred  mental  model  -  waypoint  B 


Now  that  the  student  has  been  observed  using  the  cyclic  to  affect  altitude,  we  can  infer  that  the 
student's  mental  model  shows  the  cyclic  as  the  agent  of  the  altitude,  as  shown  below.  The 
altimeter  is  shown  here  because  the  student  looked  at  it. 


■  Compare  the  actual  (simulated)  situation  with  the  expected  situation. 

■  Compare  the  student's  observed  behavior  with  the  expected  behavior. 

■  Compare  the  student’s  inferred  mental  model  with  the  expected  mental  model. 


In  this  example, 

■  the  actual  situation  may  match  the  expected  situation.  In  this  case  we  assume  no  change  in 
flight  parameters. 

■  the  observed  behavior  does  not  match  or  include  the  expected  behavior 

■  the  student's  inferred  mental  model  does  not  match  or  include  the  expected  mental  model. 
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We  can  therefore  reasonably  conclude  that  the  task  was  performed  incorrectly.  Furthermore,  we 
can  reasonably  determine  what  student  misconception  caused  the  incorrect  behavior.  We  will  now 
briefly  discuss  how  the  system  can  adapt  to  this  student. 

Adaptation 

After  the  two  steps  have  been  performed,  we  can  infer  the  following  graph  as  the  student's  mental 
model.  The  student  actually  believed  that  the  cyclic  controlled  both  altitude  and  pitch.  Figure  16 
shows  what  the  system  can  infer  about  the  student's  "actual"  mental  model.  Note  the  grayed 
portion  indicating  the  mismatch: 


Not  only  can  graph  operations  provide  a  reasonable  "best  match"  for  analysis  purposes,  we  can 
also  use  the  "difference"  between  two  graphs  to  our  benefit.  For  instance,  a  comparison  of  the 
expected  mental  model  and  the  actual  mental  model  reveals  that  they  almost  match,  with  the 
exception  of  the  shaded  portion  shown  in  this  model. 

Adaptation  of  the  instruction  is  therefore  required,  at  this  point,  for  two  purposes: 

■  address  the  misconception  that  the  cyclic  controls  altitude 

■  distinguish  between  the  operation  of  the  cyclic  and  the  collective. 

The  system  administration  module(s)  would  identify  previous  lessons  on  the  operation  of  the  cyclic 
and  operations  of  the  cyclic  vs.  the  collective  and  switch  to  those  lessons. 
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